Method and system for noise reduction with a motion compensated temporal filter

ABSTRACT

Described herein is a method and system for the reduction of noise in a video sequence. When motion is present in the video sequence, this system and method identifies motion data. With the motion data, a Motion Compensated Temporal Filter (MCTF) can apply motion compensation prior to filtering in the time domain. Temporal filtering can be performed to reduce visible noise and other detrimental artifacts.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of and claims priority to U.S.application Ser. No. 11/485,667, filed Jul. 13, 2006, entitled “Methodand System for Noise Reduction with a Motion Compensated TemporalFilter,” which claims priority to U.S. Provisional Application Ser. No.60/701,179, filed Jul. 18, 2005, entitled “Method and System for NoiseReduction with a Motion Compensated Temporal Filter,” which is herebyincorporated herein by reference in its entirety.

This application is related to the following applications, each of whichis hereby incorporated herein by reference in its entirety for allpurposes:

U.S. Provisional Patent Application Ser. No. 60/701,181, METHOD ANDSYSTEM FOR MOTION COMPENSATION, filed Jul. 18, 2005 by MacInnis;

U.S. Provisional Patent Application Ser. No. 60/701,180, METHOD ANDSYSTEM FOR VIDEO EVALUATION IN THE PRESENCE OF CROSS-CHROMAINTERFERENCE, filed Jul. 18, 2005 by MacInnis;

U.S. Provisional Patent Application Ser. No. 60/701,178, METHOD ANDSYSTEM FOR ADAPTIVE FILM GRAIN NOISE PROCESSING, filed Jul. 18, 2005 byMacInnis;

U.S. Provisional Patent Application Ser. No. 60/701,177, METHOD ANDSYSTEM FOR ESTIMATING NOISE IN VIDEO DATA, filed Jul. 18, 2005 byMacInnis;

U.S. Provisional Patent Application Ser. No. 60/701,182, METHOD ANDSYSTEM FOR MOTION ESTIMATION, filed Jul. 18, 2005 by MacInnis;

U.S. patent application Ser. No. 11/314,679, METHOD AND SYSTEM FORANALOG VIDEO NOISE REDUCTION BY BLENDING FIR AND IIR FILTERING, filedDec. 20, 2005 by Zhong; and

U.S. patent application Ser. No. 11/314,680, METHOD AND SYSTEM FORCONTENT ADAPTIVE ANALOG VIDEO NOISE DETECTION, filed Dec. 20, 2005 byZhong et al.

BACKGROUND OF THE INVENTION

The three major analog television standards are NTSC, PAL, and SECAM.The National Television System Committee developed the NTSC standard inthe U.S. in 1953. NTSC is used in United States, Canada, Japan, in mostof the American continent countries, and in various Asian countries.NTSC runs on 525 lines/frame with a vertical frequency of 60 Hz and aframerate of 29.97 frames/sec.

The PAL (Phase Alternating Line) standard was introduced in the early1960's in Europe. It has better resolution than NTSC, with 625lines/frame. The framerate is slightly lower than that of NTSC, being 25frames/sec. PAL is used in most western European countries (exceptFrance), Australia, some countries of Africa, some countries of SouthAmerica, and in some Asian countries.

The SECAM (Sequential Color with Memory) standard was introduced in theearly 1960's. SECAM uses the same bandwidth as PAL but transmits thecolor information sequentially. SECAM runs on 625 lines/frame with aframerate of 25 fps. SECAM is used in France, former French colonies,and in former communist countries in Europe.

Analog video may be received through broadcast, cable, and VCRs. Thereception is often corrupted by noise, and therefore to improve thevisual quality, noise reduction may be needed. Various noise filtershave been utilized in video communication systems and set top boxes.However, inaccurate noise characterization, especially during sceneswith motion, can result in artifacts from the filtering that are morevisually detrimental than the original noise.

Limitations and disadvantages of conventional and traditional approacheswill become apparent to one of ordinary skill in the art throughcomparison of such systems with the present invention as set forth inthe remainder of the present application with reference to the drawings.

BRIEF SUMMARY OF THE INVENTION

Described herein are system(s) and method(s) for noise reduction,substantially as shown in and/or described in connection with at leastone of the figures, as set forth more completely in the claims.

These and other advantages and novel features of the present inventionwill be more fully understood from the following description.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of an exemplary motion compensated temporalfilter in accordance with an embodiment of the present invention;

FIG. 2 is a block diagram describing motion estimation in accordancewith an embodiment of the present invention;

FIG. 3 is a block diagram of an exemplary motion estimator in accordancewith an embodiment of the present invention;

FIG. 4 is a block diagram of an exemplary motion compensator inaccordance with an embodiment of the present invention;

FIG. 5 is a block diagram of an exemplary video encoding systemcomprising a motion compensated temporal filter in accordance with anembodiment of the present invention;

FIG. 6 is a picture of an exemplary video display device in accordancewith an embodiment of the present invention; and

FIG. 7 is a flow diagram of an exemplary method for filtering a videosequence in accordance with an embodiment of the present invention.

DETAILED DESCRIPTION OF THE INVENTION

A processor may receive a video sequence that contains noise. When thevideo sequence is static, a temporal noise filter can be applied toreduce the noise. When objects in the section begin to move, a subtleedge of a moving object can cause motion trails when temporallyfiltered. To avoid creating motion trails or other video artifacts, thenoise filter may be turned off, but when the noise is no longerfiltered, it appears to a viewer that the noise level increases. Thenoise filter may also create additional video artifacts when it isturned on and off.

For example, a picture may contain a person's face, and while the personis still, the picture may appear very clear. When the person beginstalking, the face may move slightly, and along the edge of the face,noise may appear. Since filtering noise in the video sequence withmotion may cause motion trails, motion compensation is applied withinthe filter to allow substantial noise reduction while also reducing thegeneration of new degradation.

A Motion Compensated Temporal Filter (MCTF) can apply motioncompensation prior to filtering in the time domain. An accurate motionvector may correspond to a translation over time between a current blockand a reference block having similar content. Since the reference blockmay not contain identical noise, a ratio of wanted signal to noise couldincrease when a current block and a motion compensated block arecombined.

In the example of a picture containing a person's face, a motion vectormay indicate that the edge of the person's face is one pixel to the leftof where it was in the previous picture. If the block in the previouspicture is moved one pixel to the left, temporal filtering can beperformed, motion trails are less likely to be created, and visiblenoise can be reduced.

Referring now to FIG. 1, a block diagram of an exemplary MotionCompensated Temporal Filter (MCTF) 101 is illustrated in accordance withan embodiment of the present invention. The MCTF 101 comprises a motionestimator 103, a motion compensator 105, an adaptive filter 109, and anadaptive filter controller 107. A frame buffer 111 may be a memory thatstores pixel data during processing. A block selector 113 may be ahardware or software element that directs blocks of pixel data forprocessing by the MCTF 101.

In FIG. 2, there is illustrated a video sequence comprising pictures201, 203, and 205 that can be used to describe motion estimation. Motionestimation may utilize a previous picture 201 and/or a future picture205. A reference block 207 in the previous picture 201 and/or areference block 211 in the future picture 205 may contain content thatis similar to a current block 209 in a current picture 203. Motionvectors 213 and 215 give the relative displacement from the currentblock 209 to the reference blocks 207 and 211 respectively.

A block is a set of pixels to which the motion vector applies. A 16×16block corresponds to a motion vector per macroblock. A 16×16 block ismore likely than a smaller block to cause false motion artifacts whenobjects having different motion velocities are small or spatially closetogether. The smallest size a block can be is 1×1, i.e. one pixel.

Since the sampling density of a block may not be the same in both thevertical axis and the horizontal axis, the dimensions of a block cantake different values. In a 4×3 interlaced picture with 720 pixelshorizontally and 240 pixels vertically in a field, the horizontalsampling density is approximately two and one-quarter times the verticalsampling density. A 3×1 block would appear approximately square whendisplayed. A “window” is an array of sample locations used formeasurement purposes.

FIG. 2 also illustrates an example of a scene change. In the first twopictures 201 and 203 a circle is displayed. In the third picture 205 asquare is displayed. There will be a high confidence that the pastreference block 207 can predict the current block 209, and there will bea low confidence that the future reference block 211 can predict thecurrent block 209.

Confidence may be different than the sum of absolute differences (SAD).The motion vector with the highest confidence may not be the motionvector with the lowest SAD. The lowest SAD vector may be erroneous dueto noise. Cost metrics, such as SAD, can be combined together in orderto estimate confidence.

The expected value of a metric can be subtracted from the actual metricand the difference can be scaled. This is a normalization that generatesa range of values. To combine cost metrics, each metric should have thesame zero point and the same range. The normalized metric values can beadded together. Alternatively, cost metrics that represent probabilitiesmay be multiplied together.

Confidence and other quality metrics utilized in certain embodiments ofthe present invention can be generated by the system(s), method(s), orapparatus described in METHOD AND SYSTEM FOR MOTION COMPENSATION,Attorney Docket No. 16840US01, filed Jul. 18, 2005 by MacInnis, which isincorporated herein by reference for all purposes.

Referring back to FIG. 1, the block selector 113 directs a current inputwindow 124 and one or more reference pictures 126 to the motionestimator 103 (shown with more detail in FIG. 3). Based on the input 124and 126, a motion vector generator 303 produces a plurality of candidatemotion vectors 305 that are associated with a plurality of qualitymetrics 307 such as confidence. These motion vectors 305 may beinterpolated to half-pixel or finer resolution and may be associatedwith blocks in a measurement window that is larger than the currentinput block.

Spatial Extension of the Measurement Window

The current input block can be compared to reference block(s) in anotherinput and/or output picture(s) that may be temporally located before orafter the current picture. Additionally, pixel(s) near the currentblock, in a neighborhood larger than the current block, can be comparedto reference pixel(s) in another input and/or output picture(s) that maybe temporally located before or after the current picture. The motionvectors associated with the current block and the neighboring pixels(s)define a measurement window. This measurement window can be extended bya number of pixels in each dimension. Within the measurement window, themotion estimator 103 can measure the quality of a candidate motionvector over an area larger than would be associated with the currentblock alone. This spatial extension of the measurement window can reducethe occurrence of spurious motion vectors that may arise from noiseassociated with small block sizes.

Temporal Extension of the Measurement Window

Noise may cause some incorrect motion vectors. It is common for a motionvector to go through the previous picture, the current picture, and thefuture picture forming a straight line in time and space. A higherconfidence can be associated with motion vectors that are consistentover time. Likewise, the candidate motion vectors for a current blockcan include the motion vector(s) that were used to predict the referenceblock(s). This extension of the measurement window in time can reducethe occurrence of spurious motion vectors that may arise from randomnoise.

A motion vector selector 301 selects one or more motion vectors 129. Amotion vector 129 is associated with a reference block and a currentinput block within the current input window 124.

The motion vector selector 301 can eliminate a candidate motion vectorbased on the noise level measured in the video stream. If a candidatemotion vector with low cost is not much better than the zero motionvector it may be due to high noise and not motion. If the cost of thelowest-cost motion vector is not sufficiently less than the cost of thezero motion vector, the zero motion vector may be selected. In thissituation, objects may not be moving and motion compensation withnon-zero motion vectors could make objects appear to wobble. A thresholdcan be set to define “sufficiently less”. The threshold can be based ona noise distribution statistic or determined empirically based onperceptual quality.

Noise distribution statistics utilized in certain embodiments of thepresent invention can be generated by the system(s), method(s), orapparatus described in METHOD AND SYSTEM FOR ESTIMATING NOISE IN VIDEODATA, Attorney Docket No. 16843US01, filed Jul. 18, 2005 by MacInnis,and incorporated herein by reference for all purposes.

The block selector 113 directs the reference block(s) 125 to the motioncompensator 105 (shown with more detail in FIG. 4). A referencecompensator 401 applies the motion vector(s) 129 to the referenceblock(s) 125 to generate one or more motion compensated (MC) blocks 405.

Spreading of Motion Vectors with High Confidence

A pixel with a non-zero motion vector may neighbor a pixel with a zeromotion vector. Zero motion vectors that are sporadically selected cancreate strange artifacts that appear as little twittering spots. Toremove this artifact, a motion vector with a high confidence can beapplied to neighboring blocks that have motion vectors with a lowerconfidence. This spreading of confident motion vectors fills in littlespots of bad vectors. Confident motion vectors may improve immediateneighbors and neighbors up to, for example, eight pixels away.

When only one MC block 405 is selected, the MC block 405 is output 131to the adaptive filter 109. When more than one motion vector 129 isselected, a combining circuit 403 can combine the MC blocks 405 based onquality metrics 127 associated with each motion vector 129 used in theMC block 405 generation. The combination is output 131 to the adaptivefilter 109.

The adaptive filter may be implemented as an IIR filter, and FIR filter,or a combination of an IIR and FIR filter. For example within an IIRadaptive filter 109, the MC block 405 or MC block combination 131 isscaled by a value _(MC) 117. The block selector 113 directs the currentblock 123 to the adaptive filter 109 where it is scaled by a value ₀115. The scaled blocks are combined 119 to generate a current outputblock 137.

The filter controller 107 adapts ₀ 115 and _(MC) 117 according to thequality metrics 127. A residual is the difference between the currentpicture and the motion compensated reference block. By examining thecharacteristics of the residual, the quality of motion compensation canbe determined. If there is content in the residual, the motioncompensation is imperfect. By examining the characteristics of theresidual, it may be determined that the portion of the output that ismotion compensated should be reduced by lowering _(MC).

A method of examining the characteristics of the residual may includecomparing the magnitude of the residual to a previously obtained noisedistribution. If the residual falls within the bounds of AWGN (AdditiveWhite Gaussian Noise) confidence may be considered to be high. If theresidual does not fit the noise distribution, ₀ can be adaptivelyincreased toward one, and _(MC) can be decreased as (1−₀).

The current output block 137 may be sent to the frame buffer 111. Insome configurations, the MCTF 101 may use the current output block 137as a reference block in a future processing cycle.

Noise Reduction by Fir Filtering

An FIR filter can be used to filter a video signal using a form ofaveraging. If a sequence of N blocks contains the same content anduncorrelated white Gaussian noise, the following table can give theimprovement in Signal to Noise Ratio (SNR).

N SNR improvement 1 0 dB (No Averaging) 2 3 dB 3 4.8 dB   4 6 dB 5 7 dB

The number of taps, N, may be fixed or adapted. The tap weights for anaveraging filter may be equal (e.g. 1/N). Alternatively, a lowpass FIRmay be designed with tap weights that have non-equal values.

Noise Reduction by IIR Filtering

The MCTF output 137 may also be used to predict the current input block123. A first order IIR filter can be formed by directing the motionestimator 103 and the motion compensator 105 to select and utilize amotion vector associated with a past output block 137. If the currentinput block 123 and the past output block 137 have the same content anduncorrelated white Gaussian noise, the following table can give theimprovement in Signal to Noise Ratio (SNR). The scalar value ₀ isassumed to equal 1−_(MC).

SNR improvement 1 0 dB (No Filtering) 2/3 3 dB 1/2 4.8 dB   2/5 6 dB 1/37 dB

Since a first order IIR filter only requires one reference, less memoryis required for some IIR filter designs. The confidence level 127associated with the selected motion vector can determine the IIR tapweight, _(MC).

Noise Reduction by Dynamically Blending Filters

As ₀ is made smaller, the time history of the IIR filter could bepotentially very long. A long time history may cause motion trails moreoften than an FIR filter. For example, a moving object having a subtlepattern (e.g. a rug or a couch) that is visibly recognizable in thepresence of noise may appear distorted. As a result of the IIRfiltering, a residue of either the noise or the pattern may not movewith the content, thereby creating a “dirty glass” effect. The contentis moving, and the “glass” is not.

The noise reduction in certain embodiments of the present invention canbe based on the system(s), method(s), or apparatus described in METHODAND SYSTEM FOR ANALOG VIDEO NOISE REDUCTION BY BLENDING FIR AND IIRFILTERING, Ser. No. 11/314,679, filed Dec. 20, 2005 by Zhong, andincorporated herein by reference for all purposes.

The noise detection in certain embodiments of the present invention canbe based on the system(s), method(s), or apparatus described in METHODAND SYSTEM FOR CONTENT ADAPTIVE ANALOG VIDEO NOISE DETECTION, Ser. No.11/314,680, filed Dec. 20, 2005 by Zhong et al., and incorporated hereinby reference for all purposes.

To take advantage of both the FIR and the IIR structures, referenceblocks may be selected from the previous input pictures, the futureinput pictures, and the output of the MCTF. With normalized metricvalues the motion compensator can dynamically blend an FIR filter withan IIR filter. Prior to blending, motion compensated reference blockswith a high confidence may be weighted more than those with a lowconfidence.

FIG. 5 is an exemplary block diagram of a video encoding systemcomprising a motion compensated temporal filter 101 in accordance withan embodiment of the present invention. If analog video is to bedigitally encoded, a motion processor 507 can use the motion data fromthe MCTF 101 to generate an a priori measure of confidence for the videoencoder 505. The video encoder 505 may produce an encoded output 515according a wide variety of standards.

The combination of digital encoding and MCTF in certain embodiments ofthe present invention can be based on the system(s), method(s), orapparatus described in METHOD AND SYSTEM FOR MOTION ESTIMATION, AttorneyDocket No. 16844US01, filed Jul. 18, 2005 by MacInnis, and incorporatedherein by reference for all purposes.

In cases where the analog video 509 is interlaced, a deinterlacer 501may be used to generate a progressive input 511 to the MCTF 101. A noiseestimator 503 may be utilized to generate noise distribution statistics513 for the MCTF 101.

FIG. 6 is a picture of an exemplary video display device 601 containingan MCTF 101 in accordance with an embodiment of the present invention.This video display device may be a set top box that receives noisyanalog video. The MCTF 101 reduces noise prior to the display or furtherprocessing of the communicated video data.

FIG. 7 is a flow diagram 700 of an exemplary method for filtering avideo sequence in accordance with an embodiment of the presentinvention. At 701, motion is estimated in a video stream, therebygenerating a motion vector, wherein the motion vector is associated witha current block, a reference block and a confidence. The motion vectorcan be selected from a plurality of candidate motion vectors based onconfidence or another quality metric or a combination thereof. Theplurality of candidate motion vectors can include motion vectorsassociated with blocks in a neighborhood near the current block.

By utilizing the motion vector to compensate for a temporal changebetween the reference block and the current block, a motion compensatedblock is generated at 703. There can be more than one reference block.Another motion vector can be selected from the plurality of candidatemotion vectors and applied to another reference block. Multiple motioncompensated reference blocks can be dynamically scaled and combinedaccording to the confidence or another quality metric. The referenceblock(s) can be a past input block(s), a future input block(s), and/or apast output block(s).

At 705, an adaptive filter output is generated according to theconfidence, the motion compensated block and the current block. Forexample in an IIR filter design, the current block and the motioncompensated block may be scaled. The scaled motion compensated block andthe scaled current block may be combined to generate the output. Thescale factors applied to the current block and the motion compensatedblock can be adapted according to the confidence or another qualitymetric.

The embodiments described herein may be implemented as a board levelproduct, as a single chip, application specific integrated circuit(ASIC), or with varying levels of a video processing circuit integratedwith other portions of the system as separate components. An integratedcircuit may store video data in memory and use an arithmetic logiccircuit to perform MCTF processing.

The degree of integration of the video processing circuit will primarilybe determined by the speed and cost considerations. Because of thesophisticated nature of modern processors, it is possible to utilize acommercially available processor, which may be implemented external toan ASIC implementation.

If the processor is available as an ASIC core or logic block, then thecommercially available processor can be implemented as part of an ASICdevice wherein certain functions can be implemented in firmware asinstructions stored in a memory. Alternatively, the functions can beimplemented as hardware accelerator units controlled by the processor.

While the present invention has been described with reference to certainembodiments, it will be understood by those skilled in the art thatvarious changes may be made and equivalents may be substituted withoutdeparting from the scope of the present invention.

Additionally, many modifications may be made to adapt a particularsituation or material to the teachings of the present invention withoutdeparting from its scope. For example, the invention can be applied tovideo data associated with a wide variety of analog and/or digitalstandards.

Therefore, it is intended that the present invention not be limited tothe particular embodiment disclosed, but that the present invention willinclude all embodiments falling within the scope of the appended claims.

Therefore, the following is claimed:
 1. A method for the reduction ofnoise in a current video block, said method comprising: in a motioncompensated temporal filter (MCTF), generating for the current videoblock candidate motion vectors and respective confidence values, whereinthe confidence values provide an indication of accuracy of each of thecandidate motion vectors in a prediction process involving a referenceblock; selecting one of the candidate motion vectors based on anevaluation of the confidence values, the selected motion vectorassociated with a first confidence value; generating a first motioncompensated block though application of the selected motion vector to afirst reference block; and filtering the current video block and thefirst motion compensated block according to the first confidence valueto generate a current filtered block, the filtering comprising: scalingthe current video block according to a first scaling value; scaling thefirst motion compensated block according to a second scaling value; andcombining the scaled current video block with the scaled first motioncompensated block, wherein the first and second scaling values are basedon the first confidence value.